Skip to content

Tokens-database refactoring#14142

Merged
JayDi85 merged 9 commits intomagefree:masterfrom
muz:tokens_db_updates
Jan 25, 2026
Merged

Tokens-database refactoring#14142
JayDi85 merged 9 commits intomagefree:masterfrom
muz:tokens_db_updates

Conversation

@muz
Copy link
Contributor

@muz muz commented Dec 26, 2025

I had the idea of poking at systematically finding and generating tokens as new set data is released. To enable this, some understanding of the current database was needed on my side. I then saw // TODO: remove "generate" from db on the codebase and some codepaths that could be tidied up a bit...

This should be a functional no-op to the end user, but internally:

  • Cleans up and addresses that TODO, removing that unnecessary column
  • Adds some documentation on the database schema
  • Validates that items in the tokens-database.txt are as expected
  • Given that we're validating more strictly with a fairly tight regex, we can safely extract and clean up tokenType and setCode extraction
  • Also removes the no-longer user ImageFileName column

I was able to confirm that the existing data in database was consistently formatted with the same number of columns;

muz@m1 resources % cat tokens-database.txt | grep -e '^|' | sed -e 's/[a-zA-Z0-9: ,\-]//g' -e "s/'//" | sort | uniq
|||||||

I was also able to confirm what used to be the ImageFileName column was empty for all rows;

muz@m1 resources % cat tokens-database.txt | grep -e '^|'  | awk -F '|' '{print $6}' | sort | uniq 


Confirmed that the only thing reading the tokens-database.txt and in active use right now is TokenRepository.java with objects it extracts being consumed downstream in the project. The generated objects from the Repository should be consistent with before.

I did note that there is a Unit Test for this class, but it's currently marked as @Ignore with // TODO: enable test after massive token fixes so opted to not touch this right now if the test itself is not exercising value. The changes it's expecting are well beyond the scope of what this PR seeks to achieve on closing out that aforementioned TODO

@muz muz force-pushed the tokens_db_updates branch from b43f10c to b23b831 Compare December 26, 2025 21:35
@muz muz marked this pull request as draft December 26, 2025 21:52
@muz muz force-pushed the tokens_db_updates branch from b23b831 to c811bd7 Compare December 27, 2025 01:54
@muz muz marked this pull request as ready for review December 27, 2025 04:55
@muz muz marked this pull request as draft January 11, 2026 06:05
@muz
Copy link
Contributor Author

muz commented Jan 11, 2026

This needs rebasing given the flurry of changes to support ECC and ECL in recent weeks

@muz muz force-pushed the tokens_db_updates branch from d3ad6ff to 9b4b6b3 Compare January 25, 2026 01:26
@muz muz marked this pull request as ready for review January 25, 2026 06:03
@muz
Copy link
Contributor Author

muz commented Jan 25, 2026

Rebased and tested with a locally built client that token data is loaded correctly and works in-game as expected.

I think this is ready for review to ultimately address the TODO on deprecating Generate. If this is merged, I can also have a stab at updating the Wiki documentation

|TOK:M3C|Phyrexian Myr||BrudicladTelchorMyrToken|
|TOK:M3C|Sand Warrior||SandWarriorToken|
|TOK:M3C|Saproling||SaprolingToken|
|TOK:M3C|Shapeshifter|1|CribSwapShapeshifterWhiteToken|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be careful, there are wrong replacements for ShapeshifterColorlessToken. New CribSwapShapeshifterWhiteToken do not exists in master. Maybe it was part of another PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I think this one got refactored in the ECL implementation piece. Good catch!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes all unique declared Token class names in the txt file, and is checking that an equivalent file exists in Mage/src/main/java/mage/game/permanent/token

muz@m1 % for token in $(cat ../../../../../resources/tokens-database.txt | grep '^|TOK' | awk -F '|' '{print $5}' | sort | uniq); do if [ ! -f "$token.java" ]; then echo $token; fi; done
AnotherSpiritToken
ClueToken
KaroxBladewingToken

With those lines fixed too, I think that validates the same sort of thing you called out here where there're references to tokens where no file exists for one reason or another.

@muz muz requested a review from JayDi85 January 25, 2026 21:10
@JayDi85 JayDi85 merged commit 15cd693 into magefree:master Jan 25, 2026
2 checks passed
@muz muz deleted the tokens_db_updates branch January 25, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants